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Most states are using a multi-tiered system of supports (MTSS) or response to intervention (RTI) 
framework for improving the quality of instruction for all students and addressing the needs of students 


at risk for poor learning outcomes. Supporting educators in using tools to calibrate their MTSS/RTI 
implementation to the state’s expected practices is one way states can help districts and schools acquire 
and use data to improve MTSS/RTI implementation. This report describes the features of 31 tools that 
21 states developed or adapted to assess key MTSS/RTI practices that are informed by the research 
literature. Most tools assessed broad practices for implementing MTSS/RTI, such as whether schools 
administer assessments for students in need of intervention. Fewer tools assessed more specific practices 
for implementing MTSS/RTI, such as whether schools are expected to administer universal screening 
twice a year with all students. The findings can serve as a resource for state education officials who are 
interested in selecting or adapting a tool to assess MTSS/RTI implementation and in learning how other 
states support districts and schools in using these tools. 


This descriptive study of the tools and approaches states are using to assess implementation of multi-tiered 
system of supports (MTSS) and response to intervention (RTI) frameworks in schools addresses a pressing need 
for the Tennessee Department of Education (TDOE; see box 1 for definitions of key terms). TDOE officials are 
seeking ways to improve early literacy outcomes through the state’s Response to Instruction and Intervention 
(RTI?) framework. In 2015 only 43 percent of Tennessee students in grade 3 were reading proficiently. Tennessee’s 
goal is to have at least 75 percent of grade 3 students reading proficiently by 2025. The state’s RTI* framework, 
which aims to prevent reading problems from escalating, is used to inform decisions about students’ eligibility for 
special education services based on their limited responsiveness to evidence-based interventions. TDOE officials 
want to know more about how school staff are implementing RTI? practices to understand whether such practic- 
es lead to improved reading outcomes. 


To learn more about RTI* implementation in schools, TDOE and the Regional Educational Laboratory (REL) Appa- 
lachia’s research and technical support experts are developing a tool to assess RTI* implementation. To guide 
the development of this tool, TDOE officials want to know how other states are assessing implementation of 
RTI practices. Although the partnership focuses primarily on early literacy, TDOE officials are also interested in 
learning about tools for multiple academic areas (literacy and math) and grade 
levels (elementary and secondary). Therefore, this study examined the tools and 


approaches other states are using to assess MTSS/RTI implementation. For additional information, 
including background 


The term MTSS/RTI is used here to reflect current policies and practices. To Pe a wecneicdl 


ensure that the study team examined the appropriate implementation assess- 
ment tools, the study needed to account for the inconsistent use of terminolo- 
gy across states to refer to tiered systems of support that address academics. 
Because MTSS and RTI share many core features (see box 1), practitioners may 


methods, and supporting 
analyses, access the report 
appendices at 
https://go.usa.gov/xdHvJ. 
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use the terms interchangeably (National Center on Response to Intervention, 2015, n.d.). Some states use the 
term RT/ to refer to tiered supports that address academics only, whereas other states use the term R7/ or the 
term MTSS to refer to an integrated system of supports that addresses both academics and behavior' (Arden, 
Gandhi, Zumeta Edmonds, & Danielson, 2017; Zumeta Edmonds, 2016). Because the current study focuses on 
tools that assess practices to improve academic outcomes, tools that address only behavior were excluded. 


Box 1. Key terms 


Assessment tool. As an instrument used to assess implementation of a multi-tiered system of supports (MTSS)/response to inter- 
vention (RTI) framework, an assessment tool helps determine how far schools have progressed or advanced through the levels of 
implementation. 


Broad MTSS/RTI practices. MTSS/RTI practices that are considered to be general activities lacking explicit detail that educa- 
tors can interpret and implement in various ways. This study classifies components and subcomponents as “broad” MTSS/RTI 
practices. 


Component. The framework for key MTSS/RTI practices used in this study is hierarchical (see table 1 in the main text). At the 
highest level the framework includes four components of MTSS/RTI (administer assessments, offer multiple tiers of instruction 
and intervention, support data-based decisionmaking, and support infrastructure practices for RTI implementation). These com- 
ponents include more detailed subcomponents or specific aspects (for example, administer universal screening measures). Sub- 
components are then broken down into dimensions that help define and measure the subcomponents (for example, use reliable 
and valid screening tools). 


Expected practices. The MTSS/RTI practices that a state requires or recommends for implementing MTSS/RTI according to that 
state’s MTSS/RTI framework. 


Key MTSS/RTI practices. The activities and procedures for implementing MTSS/RTI. These practices are informed by the national 
evaluation of RTI for elementary school reading’s rationale for identifying key RTI practice guidelines (Balu et al., 2015); research 
literature (for example, Gersten et al., 2008) to fully define the practices; other state tools; and expert review (see table 1 in the 
main text; see appendix B on methods for coding and analyzing key MTSS/RTI practices). The key MTSS/RTI practices are organized 
by component, subcomponent, and dimension. The study team reviewed the descriptive and correlational research literature and 
identified practice guidelines that experts recommend, such as administering universal screening at least twice a year to all stu- 
dents, using reliable and valid assessment tools, varying instructional intensity through group size and instruction dosage, moni- 
toring tier 2 students monthly, and monitoring tier 3 students weekly. (See appendix A for a summary of MTSS/RTI key practices.) 
The team also examined the research literature to identify practices that are critical to all MTSS/RTI models, such as using data 
to make decisions about instruction and interventions, monitor student progress, align interventions to the core curriculum, and 
individualize tier 2 and 3 intervention. The team also reviewed state and national tools for assessing MTSS/RTI implementation 
to identify the practices and how each tool organizes them. Finally, experts in RTI, positive behavioral interventions and supports 
(PBIS), reading research, and data-based decisionmaking reviewed and vetted the list of key MTSS/RTI practices. 


Levels of implementation. The progressive stages of implementing an MTSS/RTI framework. Levels are typically described 
on a continuum, ranging from not having any practices in place to implementing ideal practices. Ideal implementation refers to 
implementation based on a state’s use of research and best practice evidence to define the practices with the highest likelihood 
of adjusting instruction to improve student outcomes. Levels of implementation may be described using descriptive labels (for 
example, not started, emerging/developing, operationalizing, or optimizing) or a numeric scale (for example, O, 1, or 2). 


Multi-tiered system of supports (MTSS). A multi-tiered framework that supports the early identification of students with learn- 
ing and behavioral challenges. MTSS addresses both academics and behavior, whereas RTI is concerned primarily with academics. 


1. Supplemental analyses for this study show that two-thirds of states that developed or adapted an implementation assessment tool 
reported using an RTI framework that addresses both academics and behavior (see appendix E). 
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Thus, MTSS is often used as an umbrella term that includes both RTI and PBIS. Multi-tiered generally refers to three tiers that cor- 
respond to different intensities of support. Tier 1, or schoolwide support and instruction, is provided to all students, and a majority 
of students (often 80—90 percent) respond well to this support. Tier 2 supports are more specialized interventions that generally 
meet the needs of the 5-15 percent of students who do not respond effectively to tier 1 supports. Tier 3 supports are more inten- 
sive, individualized interventions that typically meet the needs of the 1-5 percent of students who do not respond effectively to 
tier 1 or 2 supports. These percentages are considered ideal for implementation of a multi-tiered framework but vary by school 
context. The MTSS framework includes processes for screening all students, providing tiered instruction and intervention sup- 
ports, and monitoring students’ progress. MTSS is typically associated with general education and providing evidence-based pro- 
grams to all students (Burns, Jimerson, VanDerHeyden, & Deno, 2016; Zumeta Edmonds, 2016). 


Response to intervention (RTI). A multi-tiered framework to address problems early for students at risk for poor learning out- 
comes. Schools identify struggling learners through universal screening and provide multiple tiers of evidence-based instruction 
and interventions; monitor student progress and adjust the intensity and nature of those interventions, depending on a student’s 
responsiveness; and, as appropriate, help identify students with learning disabilities or other disabilities according to state and 
district guidance (Center on Response to Intervention & National Center on Intensive Intervention, 2014). 


Specific MTSS/RTI practices. MTSS/RTI practices that describe precise actions in a manner that educators interpret in the same 
way and that support implementing a practice as expected by a state (for example, specifying how to implement a practice, with 
whom, or how often). This study classifies dimensions as specific MTSS/RTI practices. 


Technical adequacy indicators. Validity and reliability are indicators that a tool is technically adequate. Validity refers to whether 
the tool assesses what it is supposed to assess. Types of validity include content validity (whether a tool properly assesses import- 
ant MTSS/RTI practices) and criterion validity (how well scores on one measure, such as tool ratings, predict scores on another 
measure, such as student test scores). Reliability refers to the extent to which a tool produces similar results under consistent con- 
ditions. One type of reliability is internal reliability, which focuses on the consistency of results across items on a tool (for example, 
tool users who responded one way to items about the administer assessment component tend to respond similarly to other items 


measuring the same component). 


Tool type. The category of instrument used to assess implementation of an MTSS/RTI framework. Types of tools include rubrics, 
rating scales, checklists, and surveys. 


As more states have put MTSS/RTI frameworks in place, studies of implementation have increased. Research 
on how district and school personnel implement an MTSS/RTI framework can inform interpretations of student 
outcomes and a framework’s success or failure. Implementation research can also inform feedback to staff of 
state education agencies, districts, and schools for continuous quality improvement (Keller-Margulis, 2012). A 
well-designed and well-applied MTSS/RTI implementation assessment tool may improve MTSS/RTI implementa- 
tion, which may in turn support better student outcomes. 


This report highlights promising practices for developing a tool to assess MTSS/RTI implementation and describes 
how other states assess implementation of key MTSS/RTI practices that are well-defined in the literature (Balu 
et al., 2015; Espin, McMaster, Rose, & Wayman, 2012; Fuchs, Deno, & Mirkin, 1984; Gersten et al., 2008; Haager, 
Klingner, & Vaughn, 2007; Jimerson, Burns, & VanDerHeyden, 2016). The report is intended to inform TDOE’s 
development and use of a tool to assess implementation of its RTI? framework. The report may also be useful for 
officials in other states who are interested in developing or adapting a tool to assess MTSS/RTI implementation. 
And it may be relevant to staff of other state education agencies, districts, and schools who are interested in 
assessing MTSS/RTI implementation to better understand implementation variation and fidelity, to identify areas 
of support needed to improve practices, and to interpret student outcomes. (For more details about the study, 
see appendix A. For information on sources, sample, and methods, see box 2 and appendix B.) 
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Box 2. Data sources, sample, and methods 


Data sources. The study team used data from two sources: a website and document review for all 50 states and the District of 
Columbia! and interviews with officials in eight states. Sources for website and document review data included publicly available 
versions of tools for assessing implementation of MTSS/RTI,? text from state education agency MTSS/RTI websites that describes 
assessment tool characteristics and approaches states use to support tool use, and documents to support tool use (for example, 
manuals and guidance documents). The study team collected the data from websites from February through May 2018 and veri- 
fied the information with state personnel. 

The study team used preliminary findings from the website and document review to select eight states for an interview to 
allow a deeper analysis of the assessment tools. The interviews were intended to elaborate on the information gathered from the 
public data sources and to provide a more nuanced understanding of selected states’ approaches to tool development, training, 
and supports that would be valuable to other states seeking to develop, adapt, or revise their tools. Interview findings are not 
intended to be generalizable. The interview data were collected in June 2018. 


Sample. The study team identified 33 states with MTSS/RTI tools and conducted an in-depth analysis of tools used by 21 of them 
(see appendix B for details). These tools met the following criteria: publicly available on a state’s website or accessible to the 
public on request, focused on a systems-level assessment of MTSS/RTI (as opposed to focused on a specific component of the 
framework), and developed by the state or adapted from existing instruments. Tools developed by organizations (for example, the 
National Center on Response to Intervention’s rubric and worksheet) were excluded if no state had modified or adapted them. 
Fifteen states used tools from organizations or other states without modification, so these were excluded from the in-depth 
analysis.? Research questions 1-3 focus on data from 21 states and their 31 assessment tools that met the inclusion criteria for 
in-depth analysis. 

The study team used an initial set of selection criteria and considered additional factors to identify a sample of eight states 
for which to study a range of tools and processes. The eight states selected for an interview had at least one assessment tool 
that met at least three of the following four criteria: the tool assessed RTI practices considered key by RTI experts (Gersten et al., 
2008) and the research literature (see appendix A); the tool included items on reading or literacy, because improving literacy was 
one of the partnership’s goals; the tool aligned with TDOE’s purpose for districts and schools to assess levels of MTSS/RTI imple- 
mentation or progress in making improvements; and there was evidence that the state used the tool for its intended purpose (for 
example, the state offered training on how to use the tool or reported results). 

Other factors considered were tool development processes (whether a state developed a tool or adapted an existing tool); 
technical merits of the tool (whether it had been pilot tested and assessed for validity and reliability); state resources (dedi- 
cated technical assistance centers, federal grants); and TDOE staff’s interest in learning more about specific states’ tools. The 
states selected for interviews were Florida, Kansas, North Carolina, Oregon, Pennsylvania, Vermont, Wisconsin, and Wyoming (see 
appendix B for details and appendix C for the interview protocol). In each selected state the study team identified a respondent 
who was most knowledgeable about how the state adapted or developed its assessment tool, trains intended users of the tool, 
and uses results from the tool. Interview data answer research question 4 and provide examples to supplement the website and 


document review findings for research questions 1-3. 


Methods. Trained study team members used structured protocol forms in Quick Base™ (a secure cloud database) to record infor- 
mation from the website and document review (see appendix B). For all 50 states and the District of Columbia the study team doc- 
umented information about the presence or absence of tools for assessing MTSS/RTI implementation; states verified the accuracy 
of this information as of May 11, 2018. The study team completed a structured protocol form for each of the 31 assessment tools 
to code information about the tool’s characteristics, development process, and MTSS/RTI practices assessed. State personnel 
examined the findings from the website and document review for accuracy and made revisions. 

Three study team members examined the website and document review data and used an iterative process to develop and 
revise coding schemes for each open-ended protocol item.* A senior study team member consulted and offered feedback on each 
iteration of the coding schemes. Each study team member served as the primary coder for one-third of the 21 items from the 
website and document review; a secondary coder reviewed all codes assigned by the primary coder. Discrepancies were resolved by 
consensus. (See appendix B for methods for coding and analyzing key MTSS/RTI practices.) For research questions 1-3, descriptive 
statistics, including frequencies, ranges, and cross-tabulations, were computed to analyze the website and document review data. 
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Two study team members conducted the interviews using a semi-structured interview protocol (see appendix C) that was 
tailored for each state according to the findings from the website and document review (see appendix B). The interviews were 
audio-recorded with the interviewee’s permission. Following the interviews, three study team members read the interview notes, 
discussed emerging themes, and identified key topics for a state analysis document. This document was the primary source for 
coding and analysis, and state personnel reviewed and confirmed the accuracy of the data (see appendix D for key takeaways from 
the state analysis documents). Two study team members collaboratively developed a coding scheme. One study team member 
served as the primary coder, and the other reviewed the coding for accuracy. 


Notes 


1. The study sample did not include five states because state personnel chose not to participate in the study or did not verify the data collected for 
their respective states. 


2. State personnel in one state verified that their state used an online digital portfolio tool that was not publicly accessible. 


3. This study is intended to inform state officials who want to develop or adapt an implementation assessment tool to customize it to their state’s MTSS 
or RTI framework, as opposed to using an existing tool without modifications. The names of tools from other organizations that states are using without 
modifications are included in appendix B. Five states both developed or adapted tools and used existing tools without modifications. 


4. For each open-ended item, two members of the study team independently reviewed all responses. One member developed an initial coding scheme, 
and then both members collaborated to revise and refine the coding scheme. For each item the primary coder for that item used the revised coding 
scheme to code the data, and the secondary coder reviewed the appropriateness of the applied codes. The two study team members discussed any 
cases where the coding scheme was difficult to apply and modified the coding scheme as needed. 


Research questions 


This study addressed four research questions to understand the tools and approaches that states are using to 
assess MTSS/RTI implementation: 


1. What types of tools do states use to assess MTSS/RTI implementation to ensure that districts and schools 
implement practices consistently and as expected? 


2. What processes do states use to develop or adapt these assessment tools? 
3. Do states use the tools that they developed or adapted to assess key MTSS/RTI practices? 


4. What approaches do the eight states selected to participate in interviews use to support districts and schools 
in using the assessment tool? 


Findings 
This section describes the findings for each research question in turn. 


Twenty-one states developed or adapted a total of 31 tools to assess implementation of a multi-tiered 
system of supports/response to intervention framework 


In addition to the 21 states that developed or adapted an existing tool to assess implementation of MTSS/RTI 
practices, 10 states used tools developed by another state or an organization without modifications, 7 states used 
tools that were in development or under revision, and 6 states were not using a tool.” 


2. Five states were dropped because state personnel chose not to participate or did not verify the data collected and summarized by the 
study team, one state developed a tool that was not publicly available, and one state was using only a general tool that did not meet the 
criteria for in-depth analysis (see appendix B). 
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Of the 21 states that developed or adapted an existing tool to assess implementation of MTSS/RTI practices, 16 
states developed a new assessment tool, and 6 states adapted an existing tool (1 state that developed a new tool 
and adapted an existing tool is counted in both groups). 


Seven of the eight state officials interviewed reported that their state had developed a tool, five of them because 
there were no existing tools that aligned with their state’s MTSS/RTI framework at the time. One interview 
respondent explained that the state had developed a tool that directly matched the state’s RTI components so 
that results from the tool would be most useful for informing implementation planning. Respondents in two other 
states that had developed new tools cited a desire for greater specificity than available tools offered. For example, 
one state developed a literacy tool to help ensure that schools were integrating research from the Report of the 
National Reading Panel as well as implementing RTI practices. 


Six states adapted existing tools: Florida’s Self-Assessment of MTSS Implementation (SAM; two states); Michi- 
gan’s Reading Tiered Fidelity Inventory (R-TFI; one state); the National Center on Response to Intervention Fidelity 
of Implementation Rubric (one state); and the School Implementation Scale (Erickson, Noonan, & Jenson, 2012; 
two states). The interview respondent in one state that had adapted Florida’s SAM reported that the adapta- 
tion included adding examples of evidence that could justify the ratings for each item on the tool, requiring tool 
administration by district MTSS personnel, developing more extensive scoring guidelines for each component to 
determine initial versus full implementation, and using an expert panel to ensure that the language in the tool 
was consistent with the state’s MTSS framework. 


Of the 21 states, 7 offered multiple tools for use by districts and schools. Of the 21 states included in the study, 14 
had one tool, 5 had two tools, 1 had three tools, and 1 had four tools that met the inclusion criteria. States used 
multiple tools for different reasons. Two states used distinct tools to focus separately on reading and math, two 
states used more than one tool to target elementary and secondary grade levels separately, two states used 
multiple tools to assess the perspectives of administrators and staff, and one state used multiple tools to assess 
implementation at both the district and school levels. 


All 21 states designed tools to assess implementation at the school level, although grade levels varied. All 21 states 
had at least one tool designed to assess implementation of MTSS/RTI practices at the school level, but the grade 
levels that the tools were designed to address varied. Ten states did not specify the grade levels addressed by 
their tools, two states specified grades K-12, one state specified preK—12, and one state specified grades K—-8. Two 
states had tools that were intended for use at the elementary school level only, and two states had separate tools 
for the elementary and secondary levels. Three states had tools designed for use at both the district and school 
levels. Interview respondents in other states explained that collecting information at the school level provided 
information about individual schools and could be aggregated to the district level. 


Since all 21 states had at least one tool designed to assess school-level implementation, all of these states includ- 
ed school personnel among those who were intended to administer the tool. More than a quarter of the 31 tools 
(29 percent) were intended for use by a team whose members would work collaboratively to arrive at consen- 
sus scores. States varied in whether they provided guidance about recommended or required team members. 
Most frequently, recommended or required team members represented a range of positions (for example, from 
general education and special education teachers to administrators and reading and math specialists). In addition 
to including school-level personnel, six states included personnel such as trained state or district facilitators or 
coaches to guide tool administration. 
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Of the 31 tools that states developed or adapted, 13 were rubrics, which describe explicit practices for 
each level of implementation and help users identify next steps toward ideal implementation 


States used four types of assessment tools: rubrics, rating scales, surveys, and checklists. 


Rubrics. Of the 31 tools, 13 (42 percent) were rubrics, which were used in nine states (43 percent). A rubric 
articulates the expectations for MTSS/RTI implementation by establishing coherence across levels and provid- 
ing clear, specific descriptions of practices for each level of implementation as schools progress toward ideal 
implementation. (See appendix D for images of a rubric and rating scale and for examples of tool types by level 
of implementation.) Rubrics describe, rather than judge, levels of MTSS/RTI implementation (Brookhart, 2013) 
at the district or school level. 

Rating scales. Of the 31 tools, 6 (19 percent) were rating scales. Five states (24 percent) used a rating scale only, 
and one state (5 percent) used a rating scale in combination with a survey. A rating scale lists ideal MTSS/RTI 
practices and allows the district-or school-based observers to indicate the degree to which they observe each 
practice at the district or school level of implementation (for example, not in place, purpose-building, infra- 
structure, initial implementation, full implementation). Rating scales differ from rubrics reviewed for this study 
in not describing each level of implementation but providing a description of the ideal practice instead. 
Checklists. Of the 31 tools, 4 (13 percent) were checklists. Among the three states that used checklists, two 
states (10 percent) used a checklist only, and one state (5 percent) used two checklists (an implementation 
readiness checklist and a process implementation checklist) in combination with surveys. A checklist presents 
a list of ideal MTSS/RTI practices with a place for district- or school-based observers to mark whether each 
practice is present or absent. Checklists do not describe levels of implementation for each ideal practice. 
Surveys. Of the 31 tools, 8 (26 percent) were surveys. Among the five states that used surveys, three states 
(14 percent) used 1 or more surveys only,? one state (5 percent) used one survey in combination with a rating 
scale, and one state (5 percent) used 2 surveys in combination with checklists. States used 2 surveys to address 
different subject areas (math and reading), grade levels (elementary and secondary), and intended users 
(administrators and instructional staff). A survey collects information from a sample of specific individuals (for 
example, teachers) who report their perceptions of whether or to what degree they use ideal MTSS/RTI prac- 
tices or to what degree these practices are present or absent in their school. A survey may include a numeric or 
categorical rating scale or yes or no response options. Survey findings represent individual self-report of imple- 
mentation practices. This differs from other types of implementation assessment tools—rubrics, rating scales, 
and checklists—that require district or school teams or representatives to assess district or school implemen- 
tation practices. 


A tool’s format and whether evidence is required to justify ratings have implications for how 
objectively the tool can assess implementation of expected practices 


Of the 31 tools, 15 (48 percent) were designed to assess the practices in place to meet specific levels of implemen- 
tation. All 13 rubrics and 2 of the 8 survey instruments were designed to assess expected MTSS/RTI practices for 
each level of implementation.* (See appendix D for details on what each tool is designed to assess and whether 
evidence to justify ratings is requested.) Because rubrics explicitly list the practices for various levels of imple- 
mentation along a continuum—such as not started, emerging/developing, operationalizing, and optimizing—tool 
administrators should be able to assess levels of implementation more objectively and determine next steps for 
progressing toward ideal implementation (U.S. Department of Education, 2014). Rubrics have the potential for 
improving implementation because they “make expectations and criteria explicit, which also facilitates feedback 
and self-assessment” (Jonsson & Svingby, 2007, p. 141). 


3. 
4. 


One state had 1 survey, and two states had 2 surveys each. 
For the two survey instruments, response options for each item articulated expected practices along an implementation continuum. 
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The remaining 16 tools (52 percent)—all 6 rating scales, all 4 checklists, and 6 of the 8 surveys—were designed to 
assess levels of implementation only against an ideal practice. Unlike rubrics, rating scales, checklists, and most 
surveys do not describe the practices for each level of implementation. Therefore, tool administrators must use 
their judgment to determine the extent to which their district or school is implementing an ideal practice. 


Of the 31 tools, 16 (52 percent) requested that users provide evidence to justify their scores, but the practice varied by 
tool type. Some states tried to bolster the objectivity of tool ratings by asking users to justify ratings with evidence. 
A majority of rubrics (77 percent) and rating scales (67 percent) and half of checklists (50 percent) requested evi- 
dence, but none of the surveys did. One state’s rating scale included prompts about school-based evidence, such as 
a description of the procedures used, to establish the quality of instruction and intervention. Other tool documents 
requested that schools submit artifacts, including coaching plans and master schedules, to substantiate ratings. 


When developing or adapting assessment tools, 12 states sought input from multiple sources, 8 
conducted pilot studies of the tools, and only 1 provided technical adequacy information 


Best practices in developing assessment tools include obtaining input from multiple perspectives and the research 
literature when conceptualizing the tool, conducting pilot or field testing to ensure that the items and instructions 
are clear to potential users, establishing technical properties to confirm that the tool is producing reliable and valid 
information, and engaging in an iterative revision process throughout tool development (Colton & Covert, 2007). 


Of the 21 states, at least 12 gained input from multiple sources during tool development. These sources included 
internal state experts and staff, such as state-funded technical assistance centers (12 states); outside experts, such 
as university researchers (11 states); school representatives (6 states); district representatives (5 states); and the 
research literature (3 states). Eight states conducted pilot tests or small-scale trial runs to examine how potential 
users might navigate the tools (for example, whether people understood the terminology).° 


Despite the importance of establishing a tool’s validity and reliability, signaling that a tool generates information 
users can trust, only one state had publicly available information about the tool’s technical adequacy.’ That state 
produced a technical manual that describes the process for addressing the tool’s validity and reliability through 
pilot testing. To establish criterion validity, the developers correlated scores on the tool with student behavioral 
and academic outcomes. To ensure content validity, indicating that the tool properly assessed important MTSS/ 
RTI practices, the state extensively reviewed related research and instruments; convened an expert review panel 
of district-, state-, and national-level experts on MTSS/RTI and positive behavioral interventions and supports 
(PBIS); and conducted cognitive interviews with district personnel. The developers examined internal reliability 
by calculating Cronbach’s alpha coefficients, a statistical measure indicating whether users respond similarly to a 
set of items assessing the same MTSS/RTI component. By including items about key MTSS/RTI practices, devel- 
opers can assure users that the tool is accurately assessing an MTSS/RTI framework. One interview respondent 
explained that limited staff capacity had precluded examining the tool’s technical adequacy but that the state had 
recently applied for a federal grant to support this type of research. 


Interview respondents typically reported using federal funding to support tool development and use 


Five of the eight state respondents interviewed reported that funding from the U.S. Department of Education’s 
Office of Special Education Programs supported development of the assessment tool. Respondents explained 


5. Four states did not provide information about the resources used to develop the tools. 

6. This count is based on publicly available information as well as information provided by states that was not publicly available during 
the data verification process. 

7. The state’s name is suppressed to meet the respondent’s request for anonymity. 
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that developing a state-specific tool and supporting its use required considerable time and resources. These five 
respondents reported that their state received Title VI Part B grant funds under the Individuals with Disabili- 
ties Education Act of 2004 (Public Law 108-446) or from the State Personnel Development discretionary grants. 
Three respondents were uncertain of their state’s funding sources. 


Some interview respondents explained that their state may fund one or more technical assistance centers to carry 
out a larger initiative focused on implementation of an MTSS, RTI, or PBIS framework. The center may be charged 
with developing and supporting use of a tool to assess implementation progress. Specific state-funded centers 
include the Florida Problem Solving/Response to Intervention Center, the Oregon Response to Instruction and 
Intervention Center, and the Wisconsin Response to Intervention Center. In funding a center, the state is signaling 
the importance of the MTSS/RTI framework by providing a dedicated resource to help school and district staff 
implement the practices and use the tools. 


Two interview respondents reported supporting a small group of districts and schools through a federal discretion- 
ary grant. One respondent explained that after the grant ended, the state no longer had the capacity and resourc- 
es to sustain the same types of supports. Staff at schools that were not part of the grant did not have access to 
the same level of technical assistance and support from coaches, and state staff did not have the capacity to 
sustain the initiative or expand it to more schools. 


Tools vary in whether they assess broad or specific practices 


Most tools (81 percent) include the broad practice of administering assessments, and fewer tools (48 percent) 
include the specific practice of administering universal screening at least twice a year. According to best practices, 
an assessment tool should include key components, subcomponents, and dimensions associated with the expec- 
tations for program implementation (Breitenstein et al., 2010). The study team assessed the 31 tools for 4 compo- 
nents, 8 subcomponents, and 24 dimensions that map to key MTSS/RTI practices described in the literature and 
that are considered promising practices (table 1). 


This study classifies components (dark blue rows in table 1) and subcomponents (light blue rows in table 1) as 
“broad” MTSS/RTI practices that capture general activities lacking explicit detail for implementing MTSS/RTI. It 
categorizes dimensions (white rows in table 1) as “specific” MTSS/RTI practices that describe more detailed and 
precise actions for implementing an MTSS/RTI framework. All 31 tools were assessed for their coverage of the key 
MTSS/RTI practices listed in table 1. 


Nearly all tools (97 percent) included at least one of the four key MTSS/RTI components (considered “broad” MTSS/ 
RTI practices). The four components are administering assessments (81 percent), offering multiple tiers of instruc- 
tion and intervention (77 percent), supporting data-based decisionmaking (74 percent), and supporting infra- 
structure practices for RTI implementation (39 percent; see table 1). Of the 31 tools, 9 (29 percent) addressed all 
four components. All tools included at least one key MTSS/RTI subcomponent, which is a practice that defines or 
describes the components in measurable ways (also considered “broad” MTSS/RTI practices). For example, most 
tools defined the assessment component by including the practices of administering universal student screening 
(90 percent) and progress monitoring (87 percent), and at least 77 percent of the tools included explicit expecta- 
tions for offering tier 1 instruction (90 percent) and tier 2 and 3 interventions (84 percent and 77 percent). 


Fewer tools included dimensions (considered “specific” MTSS/RTI practices) that describe specific practices 
within subcomponents. For example, while 90 percent of tools included the subcomponent of administering 
universal screening measures, fewer tools included dimensions that reflect more explicit practices associat- 
ed with an MTSS/RTI framework, such as screening at least twice a year (48 percent) or screening all stu- 
dents (65 percent; Deno, 2016; Fuchs et al., 1984; Fuchs, Fuchs, Hamlett, & Stecker, 1991; Gersten et al., 2008; 


REL 2020-017 9 


Table 1. Percentage of tools with each multi-tiered system of supports/response to intervention key practice, 
by component, subcomponent, and dimension 


Key practice Percent of tools 


Administer universal screening measures 90 
Establish end-of-year benchmarks 13 
Administer at least twice a year 48 
Use reliable and valid screening tools 52 
Include all students 65 
Administer progress monitoring measures 87 
Administer progress monitoring measures monthly at tier 2 and weekly at tier 3 29 
Use reliable and valid progress monitoring tools 39 


Offer tier 1 instruction 90 
Use evidence-based programs 45 
Tie to standards-based curriculum 61 
Address differentiation of instruction 68 
Offer tier 1 to all students 74 
Offer tier 2 intervention 84 
Require consideration of group size and dosage 29 
Tie to core curriculum 42 
Use evidence-based interventions 45 
Provide individualized instruction 52 
Offer tier 3 intervention 77 
Require consideration of group size and dosage 29 
Tie to core curriculum 29 
Use evidence-based interventions 32 
Provide individualized instruction 48 


Establish data rules 74 
Use tier 2 and 3 progress monitoring data to determine responsiveness to interventions 39 
Use multiple sources of data to inform decisions 42 
Use analysis of tier 2 and 3 progress monitoring data inclusive of slope of improvement or progress toward attainment of a goal 52 


Establish building-level implementation teams 87 
Allocate time for teams to meet 26 
Evaluate individual student progress using screening and progress monitoring data 29 
Use a problem-solving approach for decisionmaking 71 
Offer coaching to support implementation (for example, to understand data) 39 


MTSS/RTI is multi-tiered system of supports/response to intervention. 


Note: Dark blue rows are components, light blue rows are subcomponents, and white rows are dimensions. The percentage of tools at the component 
level refers to tools that include all subcomponents for a given component. The percentage of tools at the subcomponent level does not necessarily 
include all the dimensions. 


Source: Authors’ in-depth analysis of tools from the website and document review in 2018. 


see table 1). Similarly, to operationalize the component practice of offering multiple tiers of instruction and 
intervention, 84 percent of tools incorporated the subcomponent practice of “offer tier 2 intervention,” 
whereas only 45 percent addressed the more specific dimension of using evidence-based interventions with 
tier 2 students (Denton, Vaughn, & Fletcher, 2003; Stoiber & Gettinger, 2016). A similar pattern is evident for 
the remaining two components (support data-based decisionmaking and support infrastructure practices for 
MTSS/RTI implementation). 


A tool that defines expected practices at the dimension level describes the practice in a concrete manner that 
educators should be able to interpret in the same way and that supports implementing a practice as expected by 
a state (for example, specifying how to implement a practice, with whom, or how often). These specific practices 
reduce subjective inference in item interpretation and rating assignment, thereby increasing the likelihood of 
interrater agreement and reliability for the instrument. Thus, tools that assess specific MTSS/RTI practices at the 
dimension level are more likely to yield reliable feedback that schools and districts can use to improve MTSS/RTI 
implementation (Zimmaro, 2004). 


Yet having a well-developed tool is only a state’s first step toward understanding MTSS/RTI implementation in 
schools. The next step, offering training and supports for school and district staff, helps ensure that they imple- 
ment the tool as intended. 


The website and document review revealed that 15 of the 21 states that developed or adapted a tool 
made resources available to support tool use, while interview respondents reported using purposeful 
site selection, training, coaching, and follow-up strategies to support tool use 


States can assist schools and districts in tracking progress toward implementation of expected practices within 
an MTSS/RTI framework by providing tools for assessing implementation and by developing a plan that supports 
schools and districts in using these tools. For research question 4 on supporting districts and schools in using the 
assessment tool, interview respondents reported using competency drivers to guide their planning approach, as 
described below (Blase, Fixsen, & Duda, 2011; Fixsen, Naoom, Blase, & Friedman, 2005). According to the State 
Implementation and Scaling-up of Evidence-based Practices Center and the National Implementation Research 
Network (2017), “competency drivers are the activities to develop, improve, and sustain educator and administra- 
tor ability to put programs and innovations into practice so [that] students benefit.” 


Some state interview respondents reported purposefully selecting sites in the initial roll-out for training on how to 
use the tool. When implementing new practices, such as using an implementation assessment tool, getting staff 
on board and building a culture of working together are paramount to success (Fixsen et al., 2005). Respondents 
reported that selecting sites for the initial roll-out was an important consideration, and different states used dif- 
ferent processes to select sites purposefully. 


One interview respondent explained that, to gradually reach all districts, the state systematically rolled out the 
tool over five years using a cohort model. For cohort 1, state MTSS personnel used data from a readiness instru- 
ment to select a small group of districts that seemed the most ready to implement MTSS. For cohort 2, they 
selected districts that were slightly less ready, along with districts that were the least ready, in an effort to provide 
the least ready districts with models of districts that were likely to implement the tool successfully. Cohorts 3 and 
4 covered the remaining districts of regular public schools, and cohort 5 included charter schools. This approach 
enabled the state to strategically and gradually roll out the tool to all districts. 


Two interview respondents reported that their states required districts to apply to participate in training. For 
example, one respondent explained that the application included forms and interviews to assess a district’s 


REL 2020-017 11 


readiness. The application process also gave the state MTSS/RTI team an opportunity to prepare districts for 
what the training would entail and the anticipated time commitment. 


Interview data indicate that states varied in whether training targeted regional or district personnel to support 
school staff in using the tool. Improving MTSS/RTI practices requires training, both to clarify the underlying ratio- 
nale for using the implementation assessment tool and to explain how it supports schools in implementing MTSS/ 
RTI practices. Interview respondents described training personnel at either the regional or district level to help 
ensure that school-based teams understand how to use the tool. 


Two interview respondents reported that regional consultants were trained at state-funded technical assistance 
centers and were then responsible for supporting school teams in using the tools and the resulting data. One 
respondent explained that the state used a limited number of regional trainers to make the trainings consistent 
and support accurate implementation of the state’s MTSS/RTI model. In comparison, three respondents report- 
ed that state MTSS/RTI staff (from either a state-funded technical assistance center or the state department of 
education) provided training to a district-level facilitator or coach, who then supported school teams in using 
the tools. One respondent commented that the state used a facilitator to ensure that someone had the requisite 
content knowledge and interpersonal skills to support completion of the team-based assessment. Three other 
respondents described how state MTSS/RTI personnel worked directly with districts and schools to support tool 
use. 


Five interview respondents reported having state-level coaches who supported too! use. Most new skills can be 
introduced during training, but multiple opportunities to practice and receive feedback are critical for mastery. 
Guidance from a coach is recommended to support schools in implementing an MTSS/RTI framework (Freeman, 
Sugai, Simonsen, & Everett, 2017; March, Castillo, Batsche, & Kincaid, 2016). 


Two interview respondents reported that districts and schools were required to commit to two or more years 
of training and subsequent coaching. One respondent described how a state-funded technical assistance center 
worked with districts by focusing on elementary school literacy for four years, middle school literacy for three 
years, and elementary school math for two years. Coaches at the state-funded center were responsible for input- 
ting the tool data into the center’s online dashboard and reviewing the data with district and building leader- 
ship teams to support action planning. Another respondent explained that school personnel used the tools with 
support from a state coach for the first three to four years and that MTSS/RTI coaches monitored districts’ data 
and guided schools in problem-solving, as needed. 


Six interview respondents reported following up with communication processes and strategies to maintain imple- 
mentation. The respondents reported on how their state followed up with staff who had been trained and what 
types of communication processes and tools they used for maintaining contact to encourage continuing use of 
the implementation tools. These processes included using online data input and scoring systems and a program 
for recognizing schools that were making progress toward full implementation of an MTSS/RTI framework. 


One interview respondent explained that the state-funded technical assistance center sent districts a link each 
year to complete the tool online, which also allowed the center to monitor which schools completed it. The center 
also supported data use in decisionmaking by employing the data to create school-level charts and action plan 
templates. 


One interview respondent shared that schools were not required to use the tools because MTSS/RTI was not 
required in the state. However, to encourage use of the tool, schools could receive various degrees of recognition 
(bronze, silver, gold, or platinum), depending on the levels or tiers, content areas, and duration for which they had 
demonstrated full implementation. 
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Limitations 


Because the data collected in the website and document review included only publicly released information that 
was available on state education agency websites, the information may not be comprehensive. Analysis was 
limited to the data from the website and document review since it was outside the scope of the study to inter- 
view an official from every state to expand on that information. For the coding of key MTSS/RTI practices, the 
findings represent only the information provided on the tools themselves and may not fully reflect key practices 
in a state’s MTSS/RTI framework. Further, the study captures information only on the tools that states confirmed 
that they were using as of May 2018. 


The interview data are not generalizable because only eight state education officials were interviewed. District 
and school staff were not interviewed, so the data provide limited information about the actual day-to-day use 
and challenges of implementing MTSS/RTI assessment tools from the perspective of district and school staff. 
Rather, interview respondents mentioned broader challenges that they observed from their vantage point in 
the state department of education or state-funded technical assistance center, such as lack of state capacity to 
provide coaching to everyone (two states), complaints from tool users that the tools were cumbersome or bur- 
densome (three states), and tool users struggling with how to use the data after completing a tool (two states). 
Finally, causal inferences cannot be drawn from the study findings about the effectiveness of any specific practic- 
es in supporting MTSS/RTI implementation. 


Implications 


State officials who are seeking to use a tool for assessing MTSS/RTI implementation may find it valuable to review 
existing tools as a first step, to consider whether selecting or adapting an existing tool may meet the state’s 
needs. State officials considering tool development can look at the 21 states that had developed or adapted tools 
as of 2018 that were accessible to the public. (See appendixes B and D for the states and tools that were included 
in the in-depth analysis.) 


Although there are many examples that can inform tool development and adaptation, states might consider the 
following approaches used by the 21 states in this study in developing tools and supporting their use. (See appen- 
dix D on the proportion of key MTSS/RTI practices addressed by each tool type.) 

e Capture key MTSS/RTI practices to the extent possible, particularly specific and clearly described dimensions 
that help ensure construct and content validity and that can facilitate agreement among educators in the same 
district or school on their ratings of practices. 

e Use a format, such as a rubric, that describes explicit practices for each level of implementation so that users 
have the information to determine next steps toward ideal implementation (Jonsson & Svingby, 2007). 

e Request evidence to justify tool ratings to bolster the validity of the information collected and support consis- 
tency in the criteria used to substantiate tool ratings across users in the same district or school. 

e Pilot test to understand how intended users interact with the tools and to inform any needed modifications 
(Colton & Covert, 2007). 

e Establish technical adequacy (reliability and validity) when developing or extensively adapting a tool to confirm 
that the tool captures the information of interest accurately and consistently among intended users (Moskal & 
Leydens, 2000; Reddy & Andrade, 2010). 

e Provide training and technical assistance on how to use the tool and the data it generates (Fixsen, Blase, 
Naoom, & Wallace, 2009). 


State agency staff seeking tools with these characteristics may want to consult Minnesota’s Reading Tiered 
Fidelity Inventory, an adapted rubric that addresses all the key MTSS/RTI components and subcomponents and 


75 percent of the 24 dimensions listed in table 1. This tool also requests that users provide evidence to support 


REL 2020-017 13 


tool ratings. Further, Minnesota is currently pilot testing the tool. State agency staff who are interested in estab- 
lishing technical adequacy for their tool may want to learn more about the process that developers can use to 
examine a tool’s validity and reliability (for example, convening expert review panels and collecting preliminary 
data on use through pilot testing). 


Time, resources, and capacity are important considerations in tool development and use. Thus, state agency per- 
sonnel may also want to consider using other types of tools in place of or in addition to rubrics. For example, 
Wyoming’s MTSS Implementation Checklist for Literacy is a rating scale that addresses all the key MTSS/RTI com- 
ponents and subcomponents and 83 percent of the 24 dimensions listed in table 1 and requests that users provide 
evidence to substantiate ratings. Pennsylvania’s Using Response to Intervention for SLD Determination: School 
Building Application (K-12) is a checklist that addresses key MTSS/RTI practices (75 percent of the components, 
88 percent of the subcomponents, and 75 percent of the dimensions listed in table 1) and requests evidence. For 
state personnel interested in using a survey to gain insight into staff members’ perceptions about implementa- 
tion, New York’s Self-Assessment Tool for Rtl Readiness captures all the key MTSS/RTI components and subcom- 
ponents and 75 percent of the dimensions listed in table 1, in addition to having been pilot tested. 


Yet having an implementation assessment tool available may not be sufficient to enhance implementation of 
MTSS/RTI practices to improve student learning outcomes (McClellan, 2010; Stuhlmann, Daniel, Dellinger, Kenton, 
& Powers, 1999). Appropriate training is needed to support school personnel in administering a tool as intended, 
analyzing the data appropriately, and applying the findings to support program improvement in a sustainable way 
(Foorman, Smith, & Kosanovich, 2017). States might consider approaches reported by the interview respondents, 
such as gradually rolling out tools through purposeful site selection, requiring an application, or establishing a 
cohort model based on school readiness to use the tool effectively. Through this gradual release, states can focus 
on delivering high-quality training to fewer sites while accommodating possible resource constraints. If resourc- 
es allow, states might also consider options for providing coaching through a state-funded technical assistance 
center for MTSS/RTI implementation; funding an onsite district MTSS/RTI director to lead, implement, and eval- 
uate the initiative; or funding state- or regional-level coaches within regional or intermediate education agencies 
that may include MTSS/RTI supports. In addition to being informative for state personnel, these findings can be 
relevant for district staff, such as those without a state-level tool or those wanting to develop a tool that is more 
closely aligned with the district’s MTSS/RTI framework. 
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